Repository navigation
Add native celld KV through an EdgeZero store service on stock celld - #422
ChristianPavilonis wants to merge 22 commits into
Conversation
5426ea6 to
d475727
Compare
prk-Jr
left a comment
There was a problem hiding this comment.
PR Review
Summary
Reviewed head bb56cfe1 against the PR base (issue-410-native-store-providers @ 3733482f). The commits since the previous reviews (on f9b0f20c) only merge base-stack changes and touch none of this PR's 43 files, so every earlier finding still applies unchanged. The blocking items from @aram356's review are still open and have no author replies. That is the reason for requesting changes here; this review adds no new blocking defect of its own.
😃 Praise
- Endpoint parsing is strict about origins: it rejects
127.1, decimal-integer IPs,localhost, trailing dots, credentials in the URL and path segments, and it requires HTTPS for authenticated access (settings.rs:231).
Findings
Blocking
- 🔧 More unguarded celld test steps (
test.yml:201-205): the samerun_test_nonzero.shgap as the existing startup-suite comment, on the five--libsteps. Posted as a reply on that thread.
Non-blocking
- ⛏ An empty
[limits]table is now silently accepted (native_bindings.rs:147) - 🤔 Two isolation probes can never fail (
check_celld_scoped_isolation.sh:95,:103) - ⛏ The timeout test doesn't prove the process group is killed (
run_native_celld_integration.test.mjs:101) - ⛏ Guide: setting limits in both places is a startup error (
native-celld-stores.md:160) - ⛏ Plan doc: wrong lockfile path and personal home-directory paths (plan
:493,:34)
CI Status (head bb56cfe1)
- Hosted: everything passes except
cache-new-revision-smokeandcache-restore-smoke(Deploy actions). Both fail on a cache miss (TARGET_CACHE_HIT: false). This PR doesn't touch that workflow, and the same jobs passed on #414 in the same batch, so this looks like a cache/infrastructure problem rather than this PR. - Local:
- axum lib tests across two feature sets pass.
- Node runner tests pass (9/9).
- Single-feature celld clippy with
-D warningson 1.98.1 is clean. --lockedmetadata passes for all three lockfiles.- The guide's TOML examples pass
config validateas documented.
- Not run locally: the real-celld TLS proof and the namespace isolation script (they need the modified celld binary and Linux).
| #[serde(deny_unknown_fields)] | ||
| struct LimitsTable { | ||
| aws: AwsPreparationLimits, | ||
| aws: Option<AwsPreparationLimits>, |
There was a problem hiding this comment.
⛏ An empty [limits] table is now silently accepted
On the base branch LimitsTable.aws was required. A bare [limits] header with no [limits.aws] was rejected as malformed, and a [limits] table combined with an explicit policy was rejected as Duplicate (self.limits.is_some()).
With both fields now Option, an empty [limits] (for example a typo, or keys placed under the wrong section) parses as LimitsTable { aws: None, celld: None }. Both providers then quietly use their default policies. The duplicate check (:351-357) now looks at each provider separately, so an empty table plus an explicit policy is also accepted.
Nothing becomes unsafe, but an operator mistake is ignored instead of reported.
Fix: in validate_with_policies, reject Some(LimitsTable { aws: None, celld: None }) as malformed.
| [[ $(request -X PUT --data-binary forbidden "$base/v1/stores/config/value?key=probe") == 403 ]] | ||
| [[ $(request "$base/v1/stores/undeclared/value?key=probe") == 403 ]] | ||
| [[ $(request "$base/shutdown") == 404 ]] | ||
| if curl --noproxy '*' --max-time 1 -sf "http://10.203.0.1:$internal/state" >/dev/null; then |
There was a problem hiding this comment.
🤔 This probe can't fail, whatever the isolation
celld starts its internal listener on --internal-listen 127.0.0.1:0 (line 38), and the port is parsed from celld internal listening on 127.0.0.1:... (line 53). Nothing ever listens on 10.203.0.1:$internal, so this curl fails whether or not the namespaces isolate anything. Only the loopback probe at line 98 and the private PID namespace actually test isolation.
The procfs loop (lines 103-107) has a similar problem: it matches only comm == celld, so a CELLD_BIN with any other basename (for example celld-scoped) passes automatically.
CI doesn't run this script, so this is non-blocking. The plan and guide do cite its result as isolation evidence, though.
Suggestion: add a control step that shows the probed address is reachable from a non-isolated namespace (or bind the internal listener to the veth address for that check), and match the server by PID or executable path rather than by name.
| "const fs=require('node:fs');", | ||
| "const child=spawn(process.execPath,['-e','setInterval(()=>{},1000)'],{stdio:'ignore'});", | ||
| "fs.writeFileSync(process.argv[1],String(child.pid));", | ||
| "process.on('SIGTERM',()=>{child.once('exit',()=>process.exit(0));child.kill('SIGTERM');});", |
There was a problem hiding this comment.
⛏ This test passes even if the runner signals only the direct child
The runner's guarantee is that it kills the whole process group (process.kill(-child.pid, signal)), so no descendant survives. Here, though, the fixture parent handles SIGTERM by forwarding it to its own child and waiting for it to exit. If kill regressed to child.kill(signal), the grandchild would still be reaped and the test would still pass.
Fix: have the fixture parent exit on SIGTERM without forwarding the signal (process.on('SIGTERM',()=>process.exit(0))). Then only the group kill can reap the grandchild.
| | Store alias | At most 64 ASCII letters, digits, underscores or hyphens | | ||
| | Preparation / operation / connect timeout | 30 / 5 / 1 seconds | | ||
|
|
||
| `[limits.celld]` or `with_celld_preparation_limits` selects one celld policy. File and explicit policies do not merge. AWS has an independent policy and accounting. Client caches hold at most 128 authorization-scoped entries. Accounting describes logical payload and metadata quotas, not process RSS. |
There was a problem hiding this comment.
⛏ "File and explicit policies do not merge" reads as though one of them simply wins. In fact, supplying both [limits.celld] and with_celld_preparation_limits fails startup with Duplicate (native_bindings.rs:351-357), and so does a second explicit call (run_options.rs:568-570). Please say that setting both is a startup error.
The guide also never names the [limits.celld] keys (max_value_bytes, operation_timeout_ms, ...). Listing them next to the limits table would help.
|
|
||
| The user subsequently authorized committing the EdgeZero changes and opening a draft PR. The PR is stacked on #414, targeting `issue-410-native-store-providers`, not main. The companion celld branch remains local and is a separate publication dependency. | ||
|
|
||
| Pre-publication locked metadata checks found stale `examples/app-demo/Cargo.lock` and `tests/fixtures/reusable-app/Cargo.lock`. Both were refreshed offline with only the new settings-only provider package and adapter edge added. No existing registry versions changed. Both locked graphs then passed, and `cargo check --offline --locked --manifest-path tests/fixtures/reusable-app/Cargo.toml -p fixture-axum` passed in 2.78 seconds using the shared native target directory. Full demo workspace tests and all three provider WASM target builds remain unexecuted for this publication step. These consumer lockfiles are outside the source fingerprint above. |
There was a problem hiding this comment.
⛏ This cites tests/fixtures/reusable-app/Cargo.lock and --manifest-path tests/fixtures/reusable-app/Cargo.toml. The real location is crates/edgezero-test-harness/fixtures/reusable-app/, so the recorded command can't be rerun as written.
Separately, lines 34, 105, 391 and 395 embed personal /home/pav/worktrees/... paths, which are worth dropping from a shared doc.
| } | ||
| } | ||
|
|
||
| pub(crate) fn endpoint(value: &str, access: &CelldAccess) -> Result<Url, PreparationError> { |
There was a problem hiding this comment.
😃 Strict origin parsing. Comparing the raw host with the parsed IP rejects 127.1 and decimal-integer forms. It also rejects localhost, trailing dots, userinfo and path segments, and it refuses plaintext HTTP for authenticated access. That closes off a whole class of SSRF and confusion tricks before any network I/O.
Brings #275 at e9ffaa8 and the refreshed heads of #499, #460, #406, #407, #409, #414, #422 and #439. Each PR merged the one below it, so this merge has no conflicts. Two edits fit #469 to #275, as #469's open review asks: - #469's ingress JA4 test calls into_core_request with one argument. #275 now passes the Axum outbound executor through that function, so the test gets an executor handle the way the other ingress tests do. - The Axum guide names EdgeZeroAxumService, which #275 removes and check_outbound_legacy_api.sh rejects. The guide now names only AxumDevServer::with_trusted_ja4_header.
…ck celld The celld providers depended on a scoped listener compiled into a modified celld. Stock celld only exposes storage to Worker code deployed into it, so EdgeZero now ships that code: services/celld-store, a Worker plus one SQLite Durable Object per namespace, deployed with `celld deploy` onto the official release. - Each mutation applies in one SQLite transaction and records its client mutation ID. Stock celld resends a buffered Durable Object call after a pooled peer tunnel fails even when the owner already ran it; the ledger answers that resend with the first outcome. An apply-before window keeps a call held through cell recovery from applying long after the client gave up. - Responses carry a new protocol header and a store instance ID, so forked servers and reset stores fail instead of answering. - Mutations are POST /put and /delete with a fresh 128-bit ID; the client still never resends. - Remove the celld config provider and the fork-only v1 record API. - Replace the modified-celld runner and namespace isolation script with a generated-consumer proof against official celld behind a TLS proxy, and a two-node fleet proof that reproduces the resend and an owner takeover.
… tokens Review follow-ups for the store service: - A 404 counts as an absent key only when the store instance header is present. A routing 404, such as from a proxy that rewrote the path, is now malformed instead of silently reporting existing keys missing. - The Worker accepts any ASCII graphic token, including commas, and still rejects repeated Authorization headers by grammar. - The Kubernetes deploy recipe maps index.js back under src/, because ConfigMap keys cannot contain "/". - The spec no longer claims to detect a rollback to an older snapshot of the same store.
Current state: KV on official celld, no modified build
Head
15bdeef6. This PR no longer depends on a modified celld. The scoped listener compiled into celld is gone; live KV now goes through an EdgeZero-owned store service that the official celld release runs as an ordinary application.Why. Stock celld only exposes storage to Worker code deployed into it, so EdgeZero ships that code:
services/celld-store, a Worker plus one SQLite Durable Object per namespace, deployed withcelld deploy. Spec:docs/superpowers/specs/2026-11-16-stock-celld-store-service-design.md.What changed in this update
services/celld-store/(single module,no_bundle, no npm dependencies). Every mutation applies in one SQLite transaction and records the client's mutation ID. Stock celld resends a buffered Durable Object call after a pooled peer tunnel fails, even if the owner already ran it (send_innerincrates/celld/main/peer_tunnel.rs, v0.6.3); the ledger answers that resend with the first outcome. An apply-before window stops a call held through cell recovery from applying long after the client gave up.x-edgezero-store-version: 1, a store instance header on every object response, mutations asPOST /putandPOST /deletewithx-edgezero-mutation-id. Forked servers and reset stores fail instead of answering. The client still never resends.celld-configprovider and the fork-only v1 record API (recordsfeature). Records will come back on this service through the record-contract work.For review: each namespace is one Durable Object. Every write for a namespace is serialized on one node and waits for durability, that node carries all of that namespace's traffic, a node failure pauses the namespace until takeover (10 s lease by default), and sharding later would be a storage format change.
Evidence (official celld v0.6.3, binary SHA-256
eb0a8158421b69d68dd5d28f2298a7d411dde498295833deeb1cc5ce53b7eb75, release attestation verified):node --testservice, runner and fleet-script suites: 17, 9 and 4 passed.edgezero-store-celldsettings / kv / all-features: 4 / 32 / 32 passed. Axumnative-bindingslib 25,celld-kvstartup 4, mixed AWS+celld lib/startup 442 + 5 + 4 passed.cargo clippy --workspace --all-targets --all-features -D warnings, focused feature-combo Clippy,cargo fmt --check, strict rustdoc for the changed crates, dependency isolation script.node scripts/run_native_celld_integration.mjs: generated consumer againstcelld devrunning the service behind a TLS proxy, authenticated. Binary and empty values, listing, 61-second TTL expiry, restart, rejected token.node scripts/run_celld_store_fleet_proof.mjs: RustFS 1.0.1 container, two nodes. A reply dropped after the owner ran a put made celld resend it; the client got 204 and the value applied once. kill -9 of the owner, then takeover: all acknowledged writes present.Not done here: forward merges into #439 and the rest of the stack, record endpoints on the service, Trusted Server #1537 fixture changes, new RC tags, and the #437 issue text (it still lists config).
Previous description (modified-celld design, superseded)
Current review readiness
Combined head:
1c90b246af22dad4e38899c8a12a0f81cb2c6fed. This PR includes native config snapshots, ordinary KV and optional conditional/physical record operations. Therecordsfeature remains default-off and requires the additive celld record-server extension. Ordinary KV does not acquire CAS semantics.Local checks pass on the combined tree:
Fresh hosted CI and review of the enlarged scope are pending. Existing review feedback remains open; earlier approvals and receipts do not cover the added record feature. Historical checkpoints below describe only their recorded inputs. No deployment, release-tag update or main merge is included.
Current review-stack checkpoint
Publication head:
f9b0f20c6857d2e638bcfe7bf1edaa5af5425bfb. This cleanup preserves the existing feature unit and PR identity. The additional change is the completed generated-native build correction: image-only code now follows the scaffold's existing Fastly-example selection, with no request-policy change. Its focused generator/native proof is retained; fresh per-branch hosted checks are not assumed green.Review order: #275 → #406 → #407 → #409 → #414 → #422 → #439 → #454. Draft integration checkpoint: #462. New SDK/group/codec/phase development and unrelated TS work are not prerequisites or additions to this PR. Earlier notes below retain their original tested revisions and limits; they are not a whole-tree pass for this new head.
Earlier implementation and proof notes
RC refresh checkpoint, 2026-11-10
Current head:
d475727691e2a80ca2dbb324568283f63e34a870.The five celld-only donor commits are replayed onto updated #414, without obsolete hosting or diagnostic ancestry. Config/KV source selection stays typed and feature-isolated; startup snapshots are bounded and KV preserves exact keys, prefixes, continuation and expiry rules.
Feature proof passes 38 celld cases, independent settings/config/KV suites, 478 Axum unit, 6 mixed AWS startup and 4 celld startup cases, CLI/isolation/runner checks and scoped strict Clippy. Independent provider source review found no confirmed defect. Lower namespace/CI fixes are merged and all five locked fetches pass.
Ordinary KV is not conditional create/CAS/atomic RMW or identity-allocation qualification. Optional conditional/physical records provide separate Add, Overwrite and generation-match operations without changing ordinary KV. Prepared celld KV does not prove runtime connectivity/grants, and snapshots are not atomic across documents. Production identity-allocation and fleet acceptance remain separate gates.
Draft pending fresh hosted checks and the combined RC audit. Two unchanged native default-parallel deadline misses remain recorded and unclassified; serial passes and untouched-foundation parallel passes do not clear them. No timing bound was relaxed. No deployment, main merge, security-alert dismissal or review-thread resolution occurred. Historical design/review context follows.
Summary
Changes
edgezero-store-celldedgezero-adapter-axumedgezero-cliand generated Axum manifestCloses
Closes #437.
Related to #410 and stacked on #414. This is store-provider work, not the celld HTTP runtime adapter in #421.
Dependencies and draft status
issue-410-native-store-providers, currently draft Add native AWS store providers #414.f72ae21a515859570622ac9d6ca9c19cf6870b81. Records additionally require the conditional-record server extension withx-celld-record-version: 1; the original listener does not support records. The companion celld changes are not published by this PR.Test plan
cargo test --workspace --all-targets: 2,028 passed, 6 ignored. Ignored acceptance tests listed below were explicitly selected separately.cargo clippy --workspace --all-targets --all-features -- -D warnings, with focused Clippy follow-ups for the final test-only additions.cargo fmt --all -- --checkandgit diff --check.cargo check --workspace --all-targets --features "fastly cloudflare spin".wasm32-wasip2check passed; full Fastly and Cloudflare target builds were not executed.examples/app-demofull workspace tests. Both excluded consumer lockfiles passed offline locked metadata validation; reusablefixture-axumcheck passed.edgezero serve --adapter axum. Production-runner and generated native binaries were exercised instead.All Cargo verification used offline locked dependencies after reviewed lockfile updates. The workspace test runner needed stdin closed because existing consent tests assume noninteractive CI. No test was excluded to resolve that run.
The real-service command requires an explicit modified binary and uses only disposable state and dummy credentials:
The implementation plan records the tested source/binary fingerprints and proof limits. No real AWS/fleet credentials or production deployment were used.
Checklist
CLAUDE.mdconventions.edgezero_coretypes.Owned follow-up review checkpoint
ef1d4ab59427642ef26b0a30d679a068f040157e. Source tree43174231bb50d19072f33ed7a19734b5bae8c966is byte-identical to previous headbb56cfe142965bea6c997d0d2e288e2698f8ee64.